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Introduction 


This  report  extends  our  research  that  was  first  published  in  2003  (Boudreau  & 
Naegle,  2003).  At  that  time,  just  as  currently,  significant  attention  was  being  paid  to 
total  ownership  cost  (TOC).  A  number  of  initiatives  were  collected  and  shared  on  a 
TOC  website  constructed  by  the  Institute  for  Defense  Analyses  (IDA;  www.ida.org). 
Additionally,  the  DAU  Acquisition  Community  Connection  website 
(https://acc.dau. mil/CommunityBrowser.aspx?id=22509&lang=en-US)  also  contains 
useful  approaches  to  TOC  and  R-TOC.  Looking  over  the  TOC  landscape  in  2003, 
one  would  not  conclude  that  there  was  a  shortage  of  ideas  related  to  reducing  TOC. 
The  same  appears  true  today — there  are  many  useful  approaches  for  reducing  TOC, 
or  weapon  system  life-cycle  costs,  reflecting  the  increasing  anxiety  over 
skyrocketing  costs  of  ownership.  Many  aspects  of  defense  acquisition  have 
continued  to  evolve,  making  it  difficult  to  know  what  has  helped  to  control  costs  and 
what  may  have  had  the  opposite  effect  or  had  no  significant  effect.  The  following 
paragraphs  provide  a  few  examples  to  help  make  the  point. 

There  are  increased  acquisition  reviews  (Linder  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  [USD(AT&L)],  2008).  PMs  and  those  working 
in  program  offices  know  that  reviews  are  expensive  and  divert  attention  from  other 
management  activities.  Have  increased  reviews  contributed  to  increased  cost  or 
have  they  reduced  it?  Has  developmental  cost  increased  while  the  larger 
sustainment  costs  have  decreased?  Does  anyone  really  know? 

Acquisition  reforms,  launched  in  the  mid-1990s,  resulted  in  many  changes  to 
the  way  we  do  acquisition  business.  For  example,  acquisition  programs  have 
reduced  their  preparation  for  sustainment.  MIL-STD-1388-2A  and  -2B,  which 
became  obsolete  under  the  Acquisition  Reform  initiatives  of  the  1990s,  were  very 
detailed  and  for  many  years  had  guided  acquisition  logistics  planning;  they  were 
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mandatory  until  circa  1995.1  These  standards  governed  supportability  analyses  and 
served  to  inform  sustainment  planning,  but  they  were  onerous  requirements  and 
sometimes  resulted  in  analyses  that  languished  on  the  shelf  and  were  never  put  to 
use.  Did  the  discontinued  use  of  these  standards  result  in  the  de-emphasis  and  de¬ 
funding  of  rigorous  sustainment  planning,  in  turn  causing  an  increase  in  the  cost  of 
sustainment  and  a  corollary  reduction  in  warfighting  system  readiness? 

Another  Acquisition  Reform  initiative  during  the  mid-1990s  created  a  bias 
against  purchasing  technical  data  packages  (TDPs).2  Did  that  result  in  the 
avoidance  of  unnecessary  and  unneeded  TDPs,  or  might  this  initiative  have 
prevented  the  purchase  of  technical  data,  leaving  a  program  with  few  good  options 
related  to  re-buys  and  purchase  of  repair  parts?  Did  it  narrow  the  range  of  choices 
related  to  component-  and  system-level  maintenance? 

Has  performance-based  logistics  (PBL) — mandated  in  the  DoD  by  the  QDR  in 
September  2001  and  implemented  in  2002  (USD[AT&L],  2002) — reduced  the  cost  of 
sustainment  or  has  it  increased  those  costs?  Coupled  with  early  tech  data  choices, 
have  logisticians  been  forced  into  choices  that  make  sustainment  more  expensive 
throughout  the  weapon  system’s  life  cycle  (Kratz  &  Buckingham,  2010)? 

First  Gut  Question 

Have  Acquisition  Reform  and  Acquisition  Excellence  initiatives  removed  acquisition 
controls  and  opened  up  an  array  of  poor  choices  for  PMs  that  have  increased 
system  life-cycle  costs  (LCC)?  Might  well-meaning  Acquisition  Reform  and 


1  In  the  mid-1990s,  there  were  numerous  Acquisition  Reform  initiatives  intended  to  streamline  acquisition 
processes  and  reduce  cost.  One  of  these  initiatives  was  “specs  and  standards”  reform.  Many  government  specs 
were  rescinded  to  reduce  the  government  burden  and  cost  of  maintaining  specs;  in  many  cases,  the  government 
switched  to  commercial  specifications  that  were  maintained  by  various  technical  societies  or  associations.  Other 
mandatory  specs  were  rescinded  because  they  were  thought  unnecessary  or  provided  insufficient  benefit  for  the 
cost  expended.  MIL-STD  1388-2A  and  -2B  were  thought  by  some  to  fall  into  the  latter  category. 

2  Another  Acquisition  Reform  initiative  was  avoiding  the  purchase  of  technical  data  packages  in  support  of  new 
systems. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-2- 


Acquisition  Excellence  initiatives  have  offered  shortcuts  that  have  ended  badly 
(Kratz  &  Buckingham,  2010)? 

Second  Gut  Question 

Has  one  of  the  principal  problems  been  lack  of  discipline?  In  our  2003  paper 
(Boudreau  &  Naegle,  2003),  we  addressed  leadership  resolve  and  the  need  to 
speak  with  one  voice  about  affordability.  In  2003,  the  new  JCID’s  directives  did  not 
emphasize  affordability.  Today  those  directives  do  (for  example,  Chairman  of  the 
Joint  Chiefs  of  Staff  [CJCS],  2009a,  Enclosure  A,  paragraph  2-b  and  Enclosure  B, 
paragraph  3-d;  CJCS,  2009b,  Enclosure  G,  paragraph  1-d  and  Appendix  A  to 
Enclosure  G,  paragraph  16;  Weapon  System  Acquisition  Reform  Act  [WSARA], 

2009,  §  201 ).  Yet  acquisition  professionals  must  ask,  do  user  study  groups 
understand  their  emerging  system’s  slice  of  mission  area  funding  over  its  life  cycle? 
Do  users  take  ownership  control  of  these  costs  by  establishing  key  performance 
parameters  (KPPs)  or  key  system  attributes  (KSAs)  for  operations  and  support  O&S 
cost  or  system  life-cycle  cost?  Do  SoS  and  net-centric  system  PMs  understand  and 
account  for  TOC  drivers  associated  with  system  changes  (especially  software)  that 
impact  system  platforms  and  platform  changes  that  impact  overarching  systems?  Do 
materiel  developers  insist  on  clear,  unambiguous  sustainment  cost  goals  and 
establish  solid,  well-reasoned  CAIV  targets?  Do  contractors  structure  their 
developments  to  deliver  warfighting  systems  that  meet  customer  cost  constraints?  A 
dominant  problem  might  be  discipline — cost  discipline — starting  with  the  OSD  and 
Service  leadership  and  including  users,  materiel  developers,  and  contractors. 

Third  Gut  Question 

Is  ownership  cost  data  being  collected  and  placed  in  databases  that  facilitate 
analysis  and  comparison  to  ownership  cost  targets  such  that,  program  by  program, 
interested  parties  can  see  whether  DoD  programs  are  performing  within  their 
affordability  constraints?  Acquisition  leaders  must  be  able  to  measure  cost 
performance.  If  they  really  want  to  get  TOC  under  control,  O&S  cost  must  be 
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sufficiently  accurate  and  detailed  that  it  can  be  used  to  suggest  where  system, 
subsystem,  or  component  improvements  are  needed. 

Congressional  Intervention 

Interestingly,  the  questions  posed  in  the  previous  sections  appear  to  have 
been  congressional  questions,  too.  Congress  already  seems  to  have  responded  to 
an  array  of  similar  concerns,  in  its  own  unique  way.  This  is  what  the  WSARA  of  2009 
is  all  about.  This  is  what  Congress  is  addressing  in  its  changes  to  Nunn-McCurdy. 
This  is  what  motivated  Congress  to  require  certificates  at  Milestones  A  and  B  (10 
U.S.C.  §  2366a,  b).  This  appears  to  be  the  congressional  motive  in  Public  Law  1 1 1- 
84  (National  Defense  Authorization  Act,  2009),  which  institutes  product  support 
managers.  Having  witnessed  a  lack  of  cost  and  process  discipline  spanning  many 
years,  particularly  in  the  area  of  sustainment  costs,  Congress  has  acted  to  enforce 
discipline,  instituting  procedures  with  force  of  law  to  get  weapon  system  costs  under 
control. 

The  Weapon  System  Acquisition  Reform  Act  of  2009 

The  Weapon  System  Acquisition  Reform  Act  of  2009  is  a  congressional 
initiative  to  increase  rigor  in  development  of  DoD  Major  Defense  Acquisition 
Programs  (MDAPs).  The  principal  intent  seems  directed  at  controlling  the  ownership 
cost  of  the  DoD’s  warfighting  systems.  The  WSARA  advances  on  a  number  of 
different  fronts,  as  follows. 

The  WSARA  named  a  series  of  appointive  positions  in  the  Office  of  the 
Secretary  of  Defense  (SECDEF)  that  would  have  key  authorities  and  responsibilities 
in  controlling  the  acquisition  process.  One  such  position  is  the  Director  of  Cost 
Assessment  and  Program  Evaluation  (Director  CAPE),  who  has  major 
responsibilities  in  the  areas  of  cost  estimating  and  analysis,  and  providing  advice  in 
planning  PPBE,  advising  the  JROC,  and  formulating  study  guidance  used  to  conduct 
analysis  of  alternatives  of  new  major  defense  acquisition  programs.  These 
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responsibilities  place  the  Director  CAPE  in  a  position  to  provide  advice  and  direction 
related  to  the  accuracy  of  acquisition  cost  estimates  and  the  affordability  of 
acquisition  programs.  The  Director  CAPE  is  charged  by  Congress  with  ensuring  the 
accuracy  of  cost  estimation  and  cost  analysis  by  prescribing  policies  and  procedures 
specifically  related  to  acquisition  programs.  The  Director  CAPE  provides  guidance  to 
and  consults  with  Office  of  the  Secretary  of  Defense  (OSD)  leadership  and  the 
secretaries  of  the  military  departments  regarding  specific  cost  estimates  and  cost 
analyses  to  be  conducted  for  a  major  MDAP  or  major  automated  information  system 
(MAIS)  program. 

JROC 

The  WSARA  (2009)  specifically  charges  the  SECDEF  to  ensure  that  the 
JROC  is  engaged  in  consideration  of  trade-offs  among  cost,  schedule,  and 
performance  objectives  (§  201).  In  our  2003  R-TOC  report  (Boudreau  &  Naegle, 
2003),  we  noted  that  the  JROC  was  not  focused  on  TOC  and  that  the  leadership 
was  not  “speaking  with  one  voice”  (p.  49)  concerning  the  importance  of  TOC.  This 
issue  now  appears  to  have  been  addressed  as  a  matter  of  law. 

Milestone  Decision  Authority  (MDA) 

The  WSARA  mandates  that  MDA  ensure  appropriate  trade-offs  among  cost, 
schedule,  and  performance  objectives  to  increase  confidence  that  the  program  is 
affordable  (WSARA,  2009,  §  201). 

Competition  Throughout  the  Life  Cycle 

The  WSARA  identifies  10  different  approaches  that  may  be  incorporated  into 
an  MDAP  acquisition  strategy  to  ensure  competition  be  used  if  cost  effective 
(WSARA,  2009,  §  202).  The  list  includes  competitive  prototyping;  dual-sourcing; 
unbundling  of  contracts;  use  of  modular,  open  architecture  to  enable  competition  for 
upgrades;  use  of  build-to-print  approaches;  and  acquisition  of  complete  TDPs — 
along  with  several  other  approaches.  These  suggested  measures  involve 
competition  among  prime  contractors  and  also  among  subcontractors  at  such  tiers 
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as  appropriate.  The  WSARA  views  competition  as  extending  into  operations  and 
sustainment  of  MDAPs. 

The  WSARA  of  2009  Summary 

There  is  no  doubt  that  the  demands  made  in  WSARA  have  increased  the 
rigor  and  discipline  required  in  acquisition  and  will  be  reflected  in  more  careful  cost 
estimation,  increased  caution  in  reviewing  technological  maturity  before  advancing 
programs  to  the  next  acquisition  step  or  phase,  better  systems  engineering  and  test 
planning,  and  renewed  reliance  on  competition.  All  of  these  facets  have  the  potential 
to  better  control  LCC.  Conversely,  all  the  same  facets  introduce  the  potential  for 
added  bureaucracy  and  unnecessary  delay.  The  WSARA  initiatives  address  past 
shortcomings  in  MDAP  acquisitions  that  have  contributed  to  the  increase  of  LCC. 
Whether  these  initiatives  will  reduce  cost  through  better  management  or  increase 
cost  through  additional  bureaucracy  remains  to  be  seen. 

Many  other  facets  of  WSARA  are  described  in  our  201 1  paper,  Total 
Ownership  Cost — Tools  and  Discipline  (Naegle  &  Boudreau,  2011). 

National  Defense  Authorization  Act  for  Fiscal  Year  2010, 
Section  805 

The  National  Defense  Authorization  Act  for  FY2010  has  special  relevance  to 
life-cycle  cost,  as  will  be  explained.  In  this  law,  Congress  mandated  product  support 
manager  (PSM)  participation  in  MDAPs.  The  law  emphasized  that  the  PSM  works 
for  the  PM,  but  is  also  specifically  tasked  to  focus  on  product  sustainment  (O&S) 
cost.  The  PSM  is  tasked  to  balance  PBL  support  for  optimization.  He  or  she  must 
review  and  revalidate  product  support  strategies  prior  to  a  change  in  strategy  or 
every  five  years  (National  Defense  Authorization  Act,  2010,  §  805).  The 
congressional  conferees  recognized  that  product  support  encompasses  a  wide 
range  of  logistics  functions,  including  readiness,  reliability,  availability,  and  logistics 
burden  (footprint)  reduction — all  of  which  explicitly  or  implicitly  impact  ownership  cost 
(Kobren,  2010,  p.  192).  The  National  Defense  Authorization  Act  for  FY2010  very 
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apparently  established  a  position  within  the  MDAP  PM  office  that  is  responsible  for 
sustainment  cost,  to  include  reliability,  which  directly  influences  sustainment  cost. 

Duncan  Hunter  National  Defense  Authorization  Act  for  Fiscal 
Year  2009,  Section  814:  Configuration  Steering  Boards  for  Cost 
Control  Under  Major  Defense  Acquisition  Programs 

This  law  introduced  a  strong  bias  toward  limiting  design  changes  to  systems. 
Note  that  the  Service  user  representative  is  not  named  as  a  member  of  the 
Configuration  Steering  Board  (CSB).  The  presumption  may  be  that  the  user  would 
tend  to  encourage  requirements  growth  and  costly  changes.  The  CSB,  for  its  part, 
will  listen  to  the  proposed  change  and  make  the  board’s  recommendations  to  the 
program  MDA.  In  Part  2,  the  PM  is  directed  to  propose  de-scoping  options  to  reduce 
cost  and  requirements.  The  MDA  is  required  to  coordinate  changes  with  the  Joint 
Staff  and  component  requirements  officials  (i.e.,  user  representatives).  The  wording 
clearly  indicates  a  bias  against  introducing  changes  that  will  increase  cost,  or  at  the 
least  deferring  such  changes  to  a  future  block  or  increment. 

Relevant  Studies  and  Reports 

GAO/T-NSIAD-98-1  23  and  Other  GAO  Reports  on  Knowledge 
Point  Management 

Knowledge  point  management  can  be  used  to  avoid  program  delays  and  the 
additional  cost  that  accompanies  schedule  delays.  For  more  than  12  years  the  GAO 
has  advocated  the  use  of  knowledge  point  management  to  guide  development  of 
warfighting  systems  and  to  control  the  advancement  of  programs  until  those  systems 
have  demonstrated  their  readiness  to  proceed  to  the  next  step  in  the  development 
process  ( Defense  Acquisition:  Improved  Program  Outcomes,  1998).  The  three 
knowledge  points  recommended  by  the  GAO  are  described  in  the  following 
paragraphs. 
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Knowledge  Point  1  occurs  near  Milestone  B.  The  user’s  requirements  must 
be  synchronized  with  technology  that  is  mature  enough  to  support  the  endeavor, 
allow  sufficient  time  scheduled  to  succeed,  and  provide  sufficient  funding  to 
complete  the  development  (GAO,  2003,  p.  16).  This  knowledge  point  became 
relatively  better  understood  when  the  Technology  Readiness  Assessment  (TRA) 
Deskbook  was  published  in  2005  (Deputy  Under  Secretary  of  Defense  for  Science 
and  Technology  [DUSD(S&T)],  2005).  Matching  requirements  with  resources  is  a 
matter  of  discipline  and  having  the  requisite  knowledge  before  proceeding  is 
necessary  because  if  any  one  of  the  several  elements  is  absent  (such  as  the 
application  of  required  technologies  while  they  are  still  immature),  the  program  will 
likely  be  delayed  and  the  impact  on  cost  may  be  severe.  Continuing  GAO  reviews 
have  shown  that  Knowledge  Point  1  demands  enormous  discipline  that  has, 
unfortunately,  often  been  beyond  the  discipline  demonstrated  by  DoD  leadership 
over  many  years. 

Knowledge  Point  2  occurs  when  the  design  demonstrates  that  it  is  able  to 
meet  performance  requirements.  The  design  must  be  stable  (i.e. ,  90%  of  the 
engineering  drawings  must  be  complete)  and  testing  must  show  that  the  system 
performs  at  an  acceptable  level  (GAO,  2003,  p.  16).  This  point  is  verified  at  the  post- 
CDR  assessment. 

Knowledge  Point  3  occurs  when  the  system  can  be  manufactured  within  cost, 
schedule,  and  quality  targets  and  operates  reliably  (GAO,  2003,  p.  16).  In  statistical 
process  control  terms,  critical  manufacturing  processes  are  in  control  and 
consistently  producing  within  quality  standards  and  design  tolerances. 

Knowledge  point  management  is  not  new,  but  has  been  an  industry  practice. 
The  same  technique  can,  and  should,  be  applied  to  DoD  system  acquisition. 

Evolutionary  Acquisition 

The  use  of  evolutionary  acquisition  fits  conveniently  with  Knowledge  Point  1 , 
discussed  previously.  Sometimes  technology  does  not  become  mature  as  soon  as 
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hoped.  Depending  on  the  circumstances,  technological  immaturity  might  delay  a 
Milestone  B  decision  and  the  associated  program  new-start.  In  some  cases,  a 
technology  that  matures  more  slowly  than  needed  may  be  substituted  by  an 
alternative  technology  that  is  mature  and  immediately  available.  Plainly,  this  decision 
hinges  on  whether  or  not  the  developing  system  can  result  in  an  increment  of  useful 
warfighting  capability — as  determined  by  the  sponsor/user.  Even  when  this  happens, 
the  program  faces  a  difficult  path  that  requires  “extra”  milestones  that  are  exhausting 
to  program  office  staff.  Such  is  the  nature  of  evolutionary  acquisition — avoiding  one 
dilemma  and  replacing  it  with  another.  The  evolutionary  approach  places  heavy 
demands  on  a  program  office,  which  must  prepare  for  a  series  of  otherwise 
unnecessary  milestones.  Is  it  worth  it? 

The  logistics  impact  of  evolutionary  acquisition  cannot  be  ignored,  either.  A 
result  of  evolutionary  acquisition  will  either  be  multiple  configurations  or  expensive 
modifications/upgrades.  Such  cost  impacts  might  play  out  for  many  years  or  even  for 
the  lifetime  of  the  warfighting  system.  These  costs  may  be  associated  training 
issues,  repair  parts  configuration  issues,  software  patches,  and  operational  impacts. 
The  cost  of  evolutionary  acquisition  could  conceivably  approach  or  even  exceed  the 
original  cost  of  the  program  delay. 

The  right  answer  in  acquisition  depends  on  the  circumstances.  The  effect  on 
ownership  cost  should  always  be  one  of  the  metrics  used  to  select  the  best  course 
of  action. 

GAO  Report  10-717 

In  July  2010,  the  GAO  (2010)  published  Defense  Management:  DOD  Needs 
Better  Information  and  Guidance  to  More  Effectively  Manage  and  Reduce  Operating 
and  Support  Costs  of  Major  Weapon  Systems  (GAO  10-717).  This  report  painted  a 
dreary  picture  of  relevant  cost  databases.  The  GAO  found  that  important  O&S  cost- 
estimate  documents  for  aviation  systems  had  not  been  retained  and  that  there  were 
apparent  gaps  in  the  DoD’s  ability  to  capture  actual  O&S  costs  through  the  Services’ 
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Visibility  and  Maintenance  of  Operations  and  Support  Costs  (VAMOSC)  databases 
(GAO,  2010,  p.  16).  Data  in  VAMOSC  and  other  Service  information  systems  or 
sources  was  inaccurate  and  incomplete  (GAO,  2010,  pp.  16-20).  The  report  stated 
that  the  important  MDAP  system  life-cycle  cost  estimates  were  not  being  routinely 
retained  or  updated,  nor  was  there  policy  requiring  that  this  be  done.  The  GAO 
pointed  out  that  there  were  no  agreed-to  O&S  cost  elements  or  metrics  for  tracking 
and  assessing  actual  O&S  cost  performance  for  the  various  categories  of  weapon 
systems.  Additionally,  operational  costs  also  were  affected  by  unexpected  changes 
in  OPTEMPO  (specifically,  flying  hours;  GAO,  2010,  p.  22).  Although  both  those 
factors  might  upset  budget  predictions,  they  need  not  upset  performance 
predictions;  rather,  if  shown  as  “cost  per  usage,”  reasonable  comparisons  might 
show  the  weapon  system’s  performance  against  baseline  performance.  Cost  per 
mile  or  cost  per  flying  hour  or  round  fired  could  be  compared  to  early  cost  estimates, 
as-tested  costs,  and  changes  in  cost  per  year.  Such  comparisons  would  never  be 
perfect,  but  they  would  suggest  whether  a  weapon  system  was  performing  within  the 
expected  range. 

Looking  specifically  at  aviation  systems  across  the  Services,  the  GAO 
reported  that  most  systems  had  no  record  of  O&S  cost  estimates  related  to  key 
milestone  decisions.  Two  aircraft  systems,  the  Air  Force  F-22A  fighter  and  the  Navy 
F-A  18F/G,  did  have  some  recorded  O&S  cost  estimates  (GAO,  2010,  pp.  24-26). 
The  two  cited  examples  suggest  the  seriousness  of  O&S  cost-estimating  inaccuracy 
and/or  cost  growth.  F-22A  actual  cost  per  flight  hour  in  2007  was  $55,783 — 67% 
higher  than  the  $33,762  that  had  been  projected  in  the  2007  President’s  Budget. 
Similarly,  on  a  flight-hour  basis,  the  Navy  F-A  18E/F  cost  $15,346  per  flight  hour  of 
operation — 40%  higher  than  the  $10,979  predicted  in  1999. 

Institute  for  Defense  Analyses  Study:  The  Major  Causes  of 
Cost  Growth  in  Defense  Acquisition 

The  2009  Institute  for  Defense  Analyses  (IDA)  study,  led  by  Gene  Porter 
(Porter  et  al. ,  2009),  examined  1 1  MDAP  systems  that  had  exhibited  significant  cost 
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growth  between  1995  and  2006.  The  primary  causes  of  cost  growth  stemmed  from 
two  defects:  “weaknesses  in  management  visibility,  direction,  and  oversight”  and 
“weaknesses  in  initial  program  definition  and  costing”  (Porter  et  al .,  2009,  pp.  ES-6- 
ES-14),  neither  of  which  was  a  new  phenomenon.  Much  of  the  blame  for  the  first 
weakness  was  “a  general  lack  of  discipline”  (Porter  et  al.,  2009,  p.  ES-6). 

Porter  et  al.  (2009)  make  a  series  of  recommendations  that  are  intended  to 
address  the  causes  of  cost  growth  reflected  in  their  study;  their  recommendations 
are  supportive  of  the  goals  of  the  WSARA  of  2009  (Porter  et  al.,  2009,  pp.  ES-15- 
ES-18). 

DOT&E  Initiative  on  Reliability  Growth 

In  his  memorandum  State  of  Reliability,  J.  Michael  Gilmore,  the  Director  of 
Operational  Test  and  Evaluation  (DOT&E,  2010),  made  the  link  that  poor  reliability  is 
a  major  contributor  to  LCC.  The  implication  is  that  the  long-held  28-72  LCC  statistics 
could  be  altered  by  front-end  attention  to  reliability  growth.  That  is,  investing  more 
research,  development,  test,  and  evaluation  (RDT&E)  funding  in  reliability 
improvement  at  the  front  end  could  result  in  higher  reliability  components  that  would 
cost  less  to  operate,  malfunctioning  less  often.  The  remarkable  thing  here  is  that 
program  leadership  has  tried  to  improve  reliability  in  many,  if  not  all,  programs. 
Gilmore  (DOT&E,  2010)  made  reference  to  a  recently  published  reliability  standard, 
ANSI/GEIA-STD-0009,  which  should  be  employed. 

Policy  Pronouncements 

The  OSD  implemented  the  2009  version  of  the  WSARA  on  December  4, 

2010,  through  the  USD(AT&L)  publication  of  Directive  Type  Memorandum  (DTM) 
09-027  (USD[AT&L],  2009).  About  10  months  later,  on  October  21, 2010,  the 
USD(AT&L)  amended  the  original  document,  establishing  a  date  by  which  the  DoDI 
5000.02  had  to  be  revised  (USD[AT&L],  2010a). 
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Target  Affordability  and  Control  Cost  Growth  for  ACAT  I 
Programs 

Corollary  to  WSARA  implementation,  the  USD(AT&L)  published  the 
Implementation  Directive  for  Better  Buying  Power— Obtaining  Greater  Efficiency  and 
Productivity  in  Defense  Spending  (USD[AT&L],  2010b).  The  intent  of  this 
implementation  directive  was  to  reach  beyond  WSARA  mandates  to  obtain  greater 
affordability-based  decision-making  in  warfighting  system  programs.  Specifically,  its 
goal  was  to  mandate  affordability  as  a  requirement.  PMs  are  now  required  to  treat 
affordability  as  a  key  performance  parameter  (KPP)  at  Milestone  A.  The  affordability 
target  is  to  be  stated  in  two  metrics:  average  unit  acquisition  cost  and  average 
annual  operating  and  support  cost  per  unit.  These  metrics  will  be  the  basis  for  pre- 
Milestone  B  decision-making  and  systems  engineering  trade-off  analysis  to  establish 
cost  and  schedule  trade  space.  Such  a  mandate  requires  a  database  similar  to  the 
one  Roper  described  (2010,  pp.  71-73). 

Recently,  there  have  been  other  significant  directive-type  memoranda  (DTM) 
that  affect  ownership  cost  and  affordability.  Some  of  these  DTMs  are  discussed  in 
more  detail  in  our  201 1  paper,  Total  Ownership  Cost — Tools  and  Discipline  (Naegle 
&  Boudreau,  2011). 

A  Specific  Navy  Initiative:  Gate  Reviews 

The  Navy  has  instituted  a  series  of  reviews,  termed  “gate  reviews,”  to  better 
control  program  development  cost.  The  Navy  Total  Ownership  Cost  Guidebook 
(Department  of  the  Navy  [DoN],  2010;  published  concurrently  with  SECNAVINST 
5000. 2E)  depicts  a  series  of  10  gate  reviews  that  stretch  across  the  pre-acquisition 
and  acquisition  phases  and  into  the  sustainment  phase.  Each  gate  review  asks 
tailored  cost  questions  relevant  to  the  specific  life-cycle  event  (DoN,  2010,  pp.  4- 
32).  The  complete  array  of  gate  reviews  is  as  follows: 

•  Gate  1 — Initial  Capabilities  Document 

•  Gate  2 — Analysis  of  Alternatives 
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•  Gate  3 — Capability  Development  Document 

•  Gate  4 — System  Design  Specification 

•  Gate  5 — RFP  for  Engineering  and  Manufacturing  Development 
Contract 

•  Gate  6  Reviews — Specifically,  Integrated  Baseline  Review,  Post 
Critical  Design  Review,  Capability  Production  Document,  Pre-Full 
Rate  Production  Decision  Review,  and  Sustainment  Sufficiency 
Review(s) 

At  each  gate  review,  formal  design  review,  and  assessment,  programs  must 
demonstrate  progress  toward  their  affordability  initiatives,  with  strong  consideration 
in  mitigation  or  reduction  of  TOC.  The  Navy’s  intent  is  to  change  the  culture  from 
what  the  authors  of  this  working  paper  perceive  as  a  shortsighted  goal  of  obtaining 
funds  for  development  and  procurement  to  the  more  complete  perspective  of  total 
life-cycle  cost  affordability. 

Gate  Review  1 ,  which  is  intended  to  shape  the  analysis  of  alternatives  (AoA  ) 
study  analysis,  requires  consideration  of  O&S  costs  based  on  current  or  similar 
systems.  TOC  guidance  for  conducting  an  AoA  study  is  intended  to  be  sufficiently 
detailed  to  inform  and  support  the  selection  of  a  materiel  solution  from  among  the 
various  AoA  candidates. 

Intermediate  gate  reviews  are  coupled  to  existing  systems  engineering  and 
acquisition  milestone  review  points.  These  reviews  become  a  forum  to  assess 
whether  program  trade-offs  and  decisions  are  controlling  life-cycle  cost  and  whether 
the  program  is  continuing  on  the  correct  affordability  azimuth.  Each  of  the  gate 
reviews  requires  briefing  of  specific  cost  charts,  making  it  unlikely  that  cost  growth 
and  schedule  slippage  can  be  obscured. 

The  Gate  6  Sustainment  Review(s),  accomplished  post-IOC,  examine  the 
warfighting  system’s  actual  performance  data  compared  to  the  system’s  KPP 
thresholds  and  the  warfighting  system’s  actual  life-cycle  cost  compared  to  its  prior 
estimates  of  ownership  cost. 
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In  the  aggregate,  gate  reviews  provide  for  oversight  and  governance  of 
MDAP  system  developments.  In  a  wider  sense,  gate  reviews  provide  a  forum  for 
lessons  learned  regarding  TOC  while  controlling  the  affordability  of  individual 
systems — and,  hence,  the  broader  portfolios  of  warfighting  systems — throughout  the 
developmental,  production,  and  sustainment  phases  of  warfighting  systems. 

Other  Initiatives 

Controls  on  Software  Development 

Driving  the  Software  Requirements  and  Architectures  for  System 
Supportability 

While  the  tools  and  techniques  described  in  this  section  were  designed  for  the 
software  components,  they  would  be  just  as  effective  for  any  non-software 
component  as  they  are  systems  engineering  (SE)-oriented  processes.  The  systems 
engineering  process  (SEP)  focus  used  does  not  attempt  to  separate  software  from 
other  components,  so  all  system  components  would  benefit  from  using  these  tools 
and  techniques. 

Software  Supportability  Analysis 

As  with  hardware  system  components,  software  supportability  attributes  must 
be  designed  into  the  system  architecture.  Many  hardware-oriented  engineering  fields 
are  now  quite  mature,  so  that  a  number  of  supportability  attributes  would  be 
automatically  included  in  any  competent  design,  even  if  they  were  not  specified  by 
the  user  community.  For  example,  the  state  of  maturity  for  the  automotive 
engineering  field  means  that,  in  any  automotive-related  program,  there  would  be 
supportability  designs  allowing  for  routine  maintenance  of  system  filters,  lubricants, 
tires,  brakes,  batteries,  and  other  normal  wear-out  items.  There  are  few,  if  any, 
corresponding  supportability  design  attributes  that  would  be  automatically  included 
in  even  the  best  software  construct.  Virtually  all  of  the  software  supportability 
attributes  required  must  be  explicitly  specified  because  they  would  not  likely  be 
included  in  the  design  architecture  without  clearly  stated  requirements.  With 
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software,  you  get  what  you  specify  and  very  little  else.  So  how  does  one  ensure  that 
required  software  supportability  attributes  are  not  overlooked? 

Logistics  Supportability  Analysis  (LSA),  performed  extremely  early,  is  one  of 
the  keys  for  developing  the  system  supportability  attributes  needed  and  expected  by 
the  warfighter.  The  F/A  18  Super  Hornet  aircraft  was  designed  for  higher  reliability 
and  improved  ease  of  maintenance  compared  to  its  predecessors  (“F/A  18,”  201 1 ) 
because  of  warfighter  needs  for  generating  combat  power  in  the  form  of  available 
aircraft  sorties.  The  LSA  performed  on  the  F/A  18  determined  that  a  design  fostering 
higher  reliability  and  faster  maintenance  turnaround  time  (the  engines  are  attached 
to  the  airframe  at  10  locations  and  can  be  changed  in  about  20  minutes  by  a  four- 
man  team)  would  result  in  more  aircraft  being  available  to  the  commander  when 
needed.  The  concept  for  software  LSA  is  no  different,  but  implementing  sound 
supportability  analyses  on  the  software  components  has  been,  at  best,  spotty  and,  at 
worst,  completely  lacking. 

To  assist  in  effective  software  LSA,  a  focus  on  the  following  elements  is  key: 
Maintainability,  Upgradeability,  Interoperability/Interfaces,  Reliability,  and  Safety  & 
Security — MUIRS. 

Maintainability 

The  amount  of  elapsed  time  between  initial  fielding  and  the  first  required 
software  maintenance  action  can  probably  be  measured  in  hours,  not  days.  The 
effectiveness  and  efficiency  of  these  required  maintenance  actions  is  dependent  on 
several  factors,  but  the  software  architecture  that  was  developed  from  the 
performance  specifications  provided  is  critical.  The  DoD  must  influence  the  software 
architecture  through  the  performance  specification  process  to  minimize  the  cost  and 
time  required  to  perform  essential  maintenance  tasks. 

Maintenance  is  one  area  in  which  software  is  fundamentally  different  from 
hardware.  Software  is  one  of  the  very  few  components  in  which  we  know  that  the 
fielded  product  has  shortcomings,  and  we  field  it  anyway.  There  are  a  number  of 
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reasons  why  this  happens;  for  instance,  there  is  typically  not  enough  time,  funding, 
or  resources  to  find  and  correct  every  error,  glitch,  or  bug,  and  not  all  of  these  are 
worth  the  effort  of  correcting.  Knowing  this,  a  sound  plan  and  resources  must  be 
available  immediately  to  quickly  correct  those  shortcomings  that  do  surface  during 
testing  and,  especially,  those  that  arise  during  warfighting  operations.  Even  when 
the  system  software  is  operating  well,  changes  and  upgrades  in  other  interfaced 
hardware  and  software  systems  will  drive  some  sort  of  software  maintenance  action 
to  the  system  software.  In  other  words,  there  will  be  a  continuous  need  for  software 
maintenance  in  the  planned  complex  SoS  architecture  envisioned  for  net-centric 
warfare. 

Because  the  frequency  of  required  software  maintenance  actions  is  going  to 
be  much  higher  than  in  other  systems,  the  cost  to  perform  these  tasks  is  likely  to  be 
higher  as  well.  One  of  the  reasons  for  this  is  that  software  is  not  maintained  by 
“maintainers,”  as  are  most  hardware  systems,  but  is  maintained  by  the  same  type  of 
people  that  originally  developed  it — software  engineers.  These  engineers  will  be 
needed  immediately  upon  fielding,  and  a  number  will  be  needed  throughout  the 
lifespan  of  the  system  to  perform  maintenance,  add  capabilities,  and  upgrade  the 
system.  There  are  several  models  available  to  estimate  the  number  of  software 
engineers  that  will  be  needed  for  support;  planning  for  funding  these  resources  must 
begin  very  early  in  the  process.  Because  the  DoD  has  a  very  limited  capability  for 
supporting  software  internally,  early  software  support  is  typically  provided  by  the 
original  developer  and  is  included  in  the  RFP  and  proposal  for  inclusion  into  the 
contract  or  as  a  follow-on  Contractor  Logistics  Support  (CLS)  contract. 

Upgradeability 

A  net-centric  environment  composed  of  numerous  systems  developed  in  an 
evolutionary  acquisition  model  will  create  an  environment  of  almost  continuous 
change  as  each  system  upgrades  its  capabilities  over  time.  System  software  will 
have  to  accommodate  the  changes  and  will  have  to,  in  turn,  be  upgraded  to  leverage 
the  consistently  added  capabilities.  The  software  architecture  design  will  play  a 
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major  role  in  how  effectively  and  efficiently  capabilities  upgrades  are  implemented, 
so  communicating  the  known,  anticipated,  and  likely  system  upgrades  will  impact 
how  the  software  developer  designs  the  software  for  known  and  unknown  upgrades. 

Trying  to  anticipate  upgrade  requirements  for  long-lived  systems  is  extremely 
challenging  to  materiel  developers,  but  is  well  worth  their  effort.  Unanticipated 
software  changes  in  the  operational  support  phase  cost  50-200  times  the  cost  in 
early  design,  so  any  software  designed  to  accommodate  an  upgrade  that  is  never 
realized  costs  virtually  nothing  when  compared  to  changing  software  later  for  a 
capability  that  could  have  been  anticipated.  For  example,  the  Army  Tactical  Missile 
System  (ATACMS)  Unitary  was  a  requirement  to  modify  the  missile  from  warhead 
air  delivery  to  surface  detonation — that  is,  flying  the  warhead  to  the  ground.  The 
contract  award  for  the  modification  was  $1 19  million.  The  warhead  was  not  new 
technology,  nor  particularly  challenging  to  integrate  with  the  missile  body.  The  vast 
majority  of  this  cost  was  to  reengineer  the  software  to  guide  the  missile  to  the 
surface.  Had  there  been  an  upgrade  requirement  for  this  type  of  mission  in  the 
original  performance  specification,  this  original  cost  (including  potential  upgrades, 
even  if  there  were  1 0  other  upgrade  requirements  that  were  never  applied)  would 
have  been  a  fraction  of  this  modification  cost. 

Interfaces/Interoperability 

OA  design  focuses  on  the  strict  control  of  interfaces  to  ensure  the  maximum 
flexibility  in  adding  or  changing  system  modules,  whether  they  are  hardware  or 
software  in  nature.  This  presupposes  that  the  system  modules  are  known — which 
seems  logical,  as  most  hardware  modules  are  well-defined  and  bounded  by  both 
physics  and  mature  engineering  standards.  In  sharp  contrast  to  hardware,  software 
modularity  is  not  bounded  by  physics,  and  there  are  very  few  software  industry 
standards  for  modular  architecture  in  software  components.  This  is  yet  another  area 
in  which  the  software  developer  needs  much  more  information  about  operational, 
maintenance,  reliability,  safety,  and  security  performance  requirements,  as  well  as 
current,  planned,  and  potential  system  upgrades.  These  requirements,  once  well 

ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  - 17  - 

NAVAL  POSTGRADUATE  SCHOOL 


defined  and  clearly  communicated,  will  drive  the  developer  to  design  a  software 
modular  architecture  supporting  OA  performance  goals.  For  example,  if  a  system 
uses  a  Global  Positioning  System  (GPS)  signal,  it  is  likely  that  the  GPS  will  change 
over  the  life  of  the  system.  Knowing  this,  the  software  developer  creates  a 
corresponding  discrete  software  module  that  is  much  easier  and  less  expensive  to 
interface  with,  change,  and  upgrade  along  with  the  GPS  system. 

With  the  system  software  modular  architecture  developed,  the  focus  returns 
to  the  interfaces  between  hardware  and  software  modules,  as  well  as  to  the  external 
interfaces  needed  for  the  desired  interoperability  of  the  net-centric  force.  Software  is, 
of  course,  one  of  the  essential  enablers  for  interoperability  and  provides  a  powerful 
tool  for  interfacing  systems,  including  systems  that  were  not  designed  to  work 
together.  Software  performing  the  function  of  “middleware”  allows  legacy  and  other 
dissimilar  systems  to  interoperate.  Obviously,  this  interoperation  provides  a 
significant  advantage,  but  it  comes  with  a  cost  in  the  form  of  maintainability, 
resources,  and  system  complexity.  As  software  interfaces  with  other  components 
and  actually  performs  the  interface  function,  controlling  it  and  ensuring  the  interfaces 
provide  the  desired  OA  capability  become  major  software-management  and 
software-discipline  challenges. 

One  method  being  employed  by  the  DoD  attempts  to  control  the  critical 
interfaces  through  a  set  of  parameters  or  protocols  rather  than  through  active 
management  of  the  network  and  network  environment.  This  method  falls  short  on 
several  levels.  It  fails  to  understand  and  control  the  effects  of  aggregating  all  of  the 
systems  in  a  net-centric  scheme.  For  instance,  each  individual  system  may  meet  all 
protocols  for  bandwidth,  but  when  all  systems  are  engaged  on  the  network,  all 
bandwidth  requirements  are  aggregated  on  the  network — overloading  the  total 
bandwidth  available  for  all  systems.  In  addition,  members  of  the  Software 
Engineering  Institute  (SEI;  Morris,  Levine,  Meyers,  Place,  &  Plakosh,  2004)  noted, 

While  these  standards  may  present  a  step  in  the  right  direction,  they 

are  limited  in  the  extent  to  which  they  facilitate  interoperability.  At  best, 
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they  define  a  minimal  infrastructure  that  consists  of  products  and  other 
standards  on  which  systems  can  be  based.  They  do  not  define  the 
common  message  semantics,  operational  protocols,  and  system 
execution  scenarios  that  are  needed  for  interoperation.  They  should 
not  be  considered  system  architectures.  For  example,  the  C4ISR 
domain-specific  information  (within  the  JTA)  identifies  acceptable 
standards  for  fiber  channels  and  radio  transmission  interfaces,  but 
does  not  specify  the  common  semantics  of  messages  to  be 
communicated  between  C4ISR  systems,  nor  does  it  define  an 
architecture  for  a  specific  C4ISR  system  or  set  of  systems,  (p.  38) 

Clearly,  understanding  and  controlling  the  interfaces  is  critical  for  effective 
interoperation  at  both  the  system  and  SoS  levels.  The  individual  PM  must  actively 
manage  all  systems’  interfaces  impacting  OA  performance,  and  a  network  PM  must 
do  the  same  for  the  critical  network  interfaces.  Due  to  this  necessity  of  constant 
management,  a  parameters-and-protocols  approach  to  net-centric  OA  performance 
is  unlikely  to  produce  the  capabilities  and  functionality  expected  by  the  warfighter. 

Understanding  the  software  interfaces  begins  with  the  software  architecture; 
controlling  the  interfaces  is  a  unique  challenge,  encompassing  the  need  to  integrate 
legacy  and  dissimilar  systems.  This  challenge  is  exacerbated  by  the  lack  of 
software  interface  standards  within  the  existing  software  engineering  environment. 
As  stated  earlier,  the  architecture  needs  to  be  driven  through  detailed  performance 
specifications,  which  will  help  define  the  interfaces  to  be  controlled.  An  effective 
method  for  controlling  the  interfaces  is  to  intensely  manage  a  well-defined  Interface 
Control  Document  (ICD),  which  should  be  a  Contract  Data  Requirements  List 
(CDRL)  deliverable  on  any  software-intensive  or  networked  system. 

Reliability 

While  the  need  for  highly  reliable  weapon  systems  is  obvious,  the  impact  on 
total  system  reliability  of  integrating  complex  software  components  is  not  so  obvious. 
Typically,  as  system  complexity  increases,  maintaining  system  reliability  becomes 
more  of  a  challenge.  Add  the  complexity  of  effectively  networking  an  SoS  (all  of 
which  are  individually  complex)  to  a  critical  warfighting  capability  that  is  constantly 
evolving  over  time,  and  reliability  becomes  daunting. 
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Once  again,  the  software  developer  must  have  an  understanding  of  reliability 
requirements  before  crafting  the  software  architecture  and  developing  the  software 
applications.  Highly  reliable  systems  often  require  redundant  capability,  and  this 
holds  true  for  software  components  as  well.  In  addition,  software  problems  tend  to 
propagate,  resulting  in  a  degradation  of  system  reliability  over  time.  For  example,  a 
Malaysian  Airlines  Boeing  777  suffered  several  flight  control  problems,  resulting  in 
the  following:  a  near  stall  situation,  contradicting  instrument  indications,  false 
warnings,  and  difficulty  controlling  the  aircraft  in  both  autopilot  and  manual  flight 
modes.  The  problems  were  traced  to  software  in  an  air  data  inertial  reference  unit 
that  was  feeding  erroneous  data  to  the  aircraft’s  primary  flight  computer  (PFC), 
which  is  used  in  both  autopilot  and  manual  flight  modes.  The  PFC  continued  to  try  to 
correct  for  the  erroneous  data  received,  adjusting  flight  control  surfaces  in  all  modes 
of  flight,  displaying  indications  that  the  aircraft  was  approaching  stall  speed  and 
overspeed  limits  simultaneously,  and  causing  wind  shear  alarms  to  sound  close  to 
landing  (Dornheim,  2005,  p.  46).  It  is  critical  for  system  reliability  that  the  software 
developers  understand  how  outputs  from  software  applications  are  used  by 
interfaced  systems  so  that  appropriate  reliability  safeguards  can  be  engineered  into 
the  developed  software. 

Software  that  freezes  or  shuts  down  the  system  when  an  anomaly  occurs  is 
certainly  not  reliable  nor  acceptable  for  critical  weapon  systems;  yet,  these 
characteristics  are  prevalent  in  commercially  based  software  systems.  Mission 
reliability  is  a  function  of  the  aggregation  of  the  system’s  subcomponent  reliability,  so 
every  software  subcomponent  is  contributing  to  or  detracting  from  that  reliability.  The 
complexity  of  software  makes  understanding  all  failure  modes  nearly  impossible,  but 
there  are  many  techniques  that  software  developers  can  employ  when  designing  the 
architecture  and  engineering  the  applications  to  improve  software  component 
reliability.  Once  requirements  are  clearly  communicated  to  the  developers,  the 
software  can  be  engineered  with  redundancy  or  “safe-mode”  capabilities  to  vastly 
improve  mission  reliability  when  anomalies  occur.  The  key  is  identifying  the  reliability 
requirements  and  making  them  clear  to  the  software  developers. 
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Safety  and  Security 

Very  few  software  applications  have  the  required  safety  margins  associated 
with  critical  weapon  systems  used  by  warfighters  in  combat  situations — where  they 
are  depending  on  these  margins  for  their  survival.  Typically,  the  software  developers 
have  only  a  vague  idea  of  what  their  software  is  doing  and  how  critical  that  function 
is  to  the  warfighter  employing  the  weapon  system.  Safety  performance  must  be 
communicated  to  the  software  developers  from  the  beginning  of  development  so 
they  have  the  link  between  software  functionality  and  systems  safety.  For  example, 
suppose  a  smart  munition  senses  that  it  does  not  have  control  of  a  critical  directional 
component,  and  it  calculates  that  it  cannot  hit  the  intended  target.  The  next  set  of 
instructions  the  software  provides  to  the  malfunctioning  system  may  well  be  critical 
to  the  safety  of  friendly  troops,  so  software  developers  must  have  the  necessary 
understanding  of  operational  safety  to  decide  how  to  code  the  software  for  what  will 
happen  next. 

Software  safety  is  clearly  linked  with  reliability  since  software  that  is  more 
reliable  is  inherently  safer.  It  is  critical  that  the  software  developer  understands  how 
the  warfighter  expects  the  software  to  operate  in  abnormal  situations,  in  degraded 
modes,  and  when  inputs  are  outside  of  expected  values.  Much  commercially  based 
software  simply  ceases  to  function  under  these  conditions  or  gives  error  messages 
that  supersede  whatever  function  was  being  performed,  neither  of  which  is 
acceptable  in  combat  operations. 

With  software  performing  so  many  critical  functions,  there  is  little  doubt  that 
software  applications  are  a  prime  target  for  anyone  opposing  U.S.  and  Allied  forces. 
Critical  weapon  system  and  networking  software  must  be  resistant  to  hacking, 
spoofing,  mimicking,  and  all  other  manner  of  attack.  There  must  be  capabilities  for 
isolating  attacks  and  portions  of  networks  that  have  been  compromised  without 
losing  the  ability  to  continue  operations  in  critical  combat  situations.  The  software 
developer  must  know  that  all  of  these  capabilities  are  essential  before  he  or  she 
constructs  software  architectures  and  software  programs,  as  this  knowledge  will  be 
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very  influential  for  the  software  design  and  application  development.  The  SEI’s 
Quality  Attribute  Workshop  (Barbacci  et  al. ,  2003)  states,  “As  an  example,  consider 
security.  It  is  difficult,  maybe  even  impossible,  to  add  effective  security  to  a  system 
as  an  afterthought.  Component  as  well  as  communication  mechanisms  and  paths 
must  be  designed  or  selected  early  in  the  lifecycle  to  satisfy  security  requirements” 

(p.  2). 


Interoperability  challenges  are  increased  when  the  SoS  has  the  type  of 
security  requirements  needed  by  the  DoD.  Legacy  systems  and  existing  security 
protocols  will  likely  need  to  be  considered  before  other  security  architecture  can  be 
effectively  designed.  OA  capabilities  will  be  hampered  by  the  critical  need  for 
security;  both  must  be  carefully  balanced  to  optimize  system  performance  and 
security.  This  balance  of  OA  and  security  must  be  managed  by  the  DoD  and  not  the 
software  developer. 

Physical  security  schemes  and  operating  procedures  will  also  have  an  impact 
on  the  software  architecture.  For  example,  many  communication  security 
(COMSEC)  devices  need  only  routine  security  until  the  keys,  usually  software 
programs,  are  applied;  then,  much  more  stringent  security  procedures  are 
implemented.  Knowledge  of  this  security  feature  would  be  a  key  requirement  of  the 
developer;  he  or  she  must  understand  how  and  when  the  critical  software  pieces  are 
uploaded  to  the  COMSEC  device.  The  same  holds  true  for  weapon  systems  that 
upload  sensitive  mission  data  just  prior  to  launch. 

Residual  software  on  equipment  or  munitions  that  could  fall  into  enemy  hands 
presents  another  type  of  security  challenge  that  needs  to  be  addressed  during 
application  development.  For  example,  the  ATACMS  missile  air-delivers  some  of  its 
warheads,  leaving  the  missile  body  to  free  fall  to  the  surface.  It  is  very  conceivable 
that  the  body  could  be  intact  and,  of  course,  unsecured.  If  critical  mission  software 
was  still  within  the  body  and  found  by  enemy  forces,  valuable  information  might  be 
gleaned  from  knowing  how  the  system  finds  its  targets.  The  government  would 
certainly  want  the  developer  to  design  the  applications  in  a  way  that  would  make 
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anything  recovered  useless  to  the  enemy,  but  this  is  a  capability  that  is  not  intuitive 
to  software  developers  (Naegle,  2006,  pp.  17-25). 

Effective  Software  Development  Tools  Supporting  System  TOC  Analyses 

Software  Engineering  Institute’s  (SEI)  Quality  Attribute  Workshop 

(QAW) 

The  QAW  is  designed  to  help  identify  a  complete  (or  as  complete  as  possible) 
inventory  of  system  software  requirements  through  analysis  of  system  quality 
attributes.  One  of  the  intents  is  to  develop  the  derived  and  implied  requirements  from 
the  user-stated  requirements,  which  is  a  necessary  step  when  user-stated 
requirements  are  provided  in  terms  of  capabilities  needed  as  prescribed  by  the  Joint 
Capabilities  Integration  Development  System  (JCIDS)  process.  A  system’s  TOC, 
and  those  elements  that  contribute  to  TOC,  are  system  quality  attributes.  Although 
obviously  important  to  the  warfighter,  the  associated  operations  and  support, 
training/education,  and  facility  costs  are  rarely  addressed  in  much  detail  and  need  to 
be  derived  from  stated  requirements  or  augmented  with  implied  requirements 
through  the  QAW  process,  or  something  similar. 

The  QAW  helps  provide  a  facilitating  framework  and  process  designed  to 
more  fully  develop  the  derived  and  implied  requirements  that  are  critical  to  clearly 
communicate  to  potential  contractors  and  software  developers.  Including  a  robust 
LSA  process  using  the  MUIRS  focus  elements,  described  previously,  within  the 
QAW  process  will  likely  significantly  improve  requirements  analysis  for  those 
associated  TOC  elements  and  vastly  improve  the  accuracy  of  system  TOC 
projections.  While  improving  system  requirements  development,  the  QAW  is 
designed  to  work  with  another  SEI  process  called  the  Architectural  Trade-off 
Analysis  MethodologySM  (ATAMsm)  to  further  improve  the  understanding  of  the 
system  for  potential  contractors  and  software  developers. 
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SEI’s  Architectural  Trade-Off  Analysis  MethodologySM  (ATAMsm) 

The  SEI’s  ATAMsm  is  an  architectural  analysis  tool  designed  to  evaluate 
design  decisions  based  on  the  quality  attribute  requirements  of  the  system  being 
developed.  The  methodology  is  a  process  for  determining  whether  the  quality 
attributes,  including  TOC  attributes,  are  achievable  by  the  architecture  as  it  has  been 
conceived  before  enormous  resources  have  been  committed  to  that  design.  One  of 
the  main  goals  is  to  gain  insight  into  how  the  quality  attributes  trade  off  against  each 
other  (Kazman,  Klein,  &  Clements,  2000,  p.  1). 

Within  the  systems  engineering  process  (SEP),  the  ATAMsm  provides  the 
critical  requirements  loop  process,  tracing  each  requirement  or  quality  attribute  to 
corresponding  functions  reflected  in  the  software  architectural  design.  Whether 
ATAMsm  or  another  analysis  technique  is  used,  this  critical  SEP  must  be  performed 
to  ensure  that  functional-  or  object-oriented  designs  meet  all  stated,  derived,  and 
implied  warfighter  requirements.  In  complex  systems  development,  such  as  weapon 
systems,  half  or  more  than  half  of  the  total  software  development  effort  will  be 
expended  in  the  architectural  design  process.  Therefore,  DoD  PMs  must  ensure  that 
the  design  is  addressing  requirements  in  context  and  that  the  resulting  architecture 
has  a  high  probability  of  producing  the  warfighters’  JCIDS  stated,  derived,  or  implied 
requirements. 

The  ATAMsm  focuses  on  quality  attribute  requirements,  so  it  is  critical  to  have 
precise  characterizations  for  each.  To  characterize  a  quality  attribute,  the  following 
questions  must  be  answered: 

•  What  are  the  stimuli  to  which  the  architecture  must  respond? 

•  What  is  the  measurable  or  observable  manifestation  of  the  quality 
attribute  by  which  its  achievement  is  judged? 

•  What  are  the  key  architectural  decisions  that  impact  achieving  the 
attribute  requirement?  (Kazman  et  al. ,  2000,  p.  5) 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  24  - 

NAVAL  POSTGRADUATE  SCHOOL 


The  ATAMsm  scenarios  are  a  key  to  providing  the  necessary  information  to 
answer  the  first  two  questions,  driving  the  software  engineer  to  design  the 
architecture  to  answer  the  third.  This  is  a  critical  point  at  which  all  of  the  MUIRS 
elements  need  to  be  considered  and  appropriate  scenarios  developed. 

The  ATAMsm  uses  three  types  of  scenarios:  use-case  scenarios  involve 
typical  uses  of  the  system  to  help  understand  quality  attributes  in  the  operational 
context;  growth  scenarios  involve  anticipated  design  requirements,  including 
upgrades,  added  interfaces  supporting  SoS  development,  and  other  maturity  needs; 
and  exploratory  scenarios  involve  extreme  conditions  and  system  stressors, 
including  Failure  Modes  and  Effects  Criticality  Analysis  (FMECA)  scenarios  (Kazman 
et  al.,  2000,  pp.  13-15).  As  depicted  in  Figure  1,  the  scenarios  build  on  the 
foundation  provided  in  the  JCIDS  documents  and  requirements  developed  through 
the  QAW  process.  These  processes  lend  themselves  to  development  in  an 
Integrated  Product  Team  (IPT)  environment  led  by  the  user/combat  developer  and 
including  all  of  the  system’s  stakeholders.  The  IPT  products  will  include  a  set  of 
scenarios,  prioritized  by  the  needs  of  the  warfighter  for  system  capability.  The 
prioritization  process  provides  a  basis  for  architecture  trade-off  analyses.  When  fully 
developed  and  prioritized,  the  scenarios  provide  a  more  complete  understanding  of 
requirements  and  quality  attributes  in  context  with  the  operation  and  support 
(including  all  of  the  MUIRS  elements)  of  the  system  over  its  life  cycle.  A  more 
complete  understanding  of  the  system’s  TOC  elements  should  emerge  from  this 
type  of  analysis. 
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Figure  1.  QAW  &  ATAMsm  Integration  Into  Software  Life-Cycle  Management 

Just  as  the  QAW  process  provides  a  methodology  supporting  the  RFP, 
source-selection  activities,  and  the  Software  Specification  and  System 
Requirements  Reviews  (SSR  and  SRR),  the  ATAMsm  provides  a  methodology 
supporting  design  analyses,  test  program  activities,  and  the  System  Functional  and 
Preliminary  Design  Reviews  (SFR  and  PDR).  The  QAW  and  ATAMsm  methodologies 
are  probably  not  the  only  effective  methods  supporting  software  development  efforts, 
but  they  fit  particularly  well  with  the  DoD’s  goals,  models,  and  SEP  emphasis.  The 
user/combat  developer  (blue  arrow  block  in  Figure  1 )  is  kept  actively  involved 
throughout  the  development  process — providing  key  insights  the  software  developer 
needs  to  successfully  develop  warfighter  capabilities  in  a  sustainable  design  for 
long-term  effectiveness  and  suitability.  The  system  development  activities  are 
conducted  with  superior  understanding  and  clarity,  reducing  scrap  and  rework,  and 
saving  cost  and  schedule.  The  technical  reviews  and  audits  (part  of  the  DoD’s 
overarching  SEP)  are  supported  with  methodologies  that  enhance  both  the  visibility 
of  the  necessary  development  work  as  well  as  the  progress  toward  completing  it. 
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One  of  the  main  goals  in  analyzing  the  scenarios  is  to  discover  key 
architectural  decision  points  that  pose  risks  for  meeting  quality  requirements. 
Sensitivity  points  are  determined,  such  as  real-time  latency  performance  shortfalls  in 
target  tracking.  Trade-off  points  are  also  examined  so  that  TOC  impacts  resulting 
from  proposed  trade-offs  can  be  analyzed.  The  SEI  explained,  “Trade-off  points  are 
the  most  critical  decisions  that  one  can  make  in  an  architecture,  which  is  why  we 
focus  on  them  so  carefully”  (Kazman  et  al.,  2000,  p.  23). 

The  ATAMsm  provides  an  analysis  methodology  that  complements  and 
enhances  many  of  the  key  DoD  acquisition  processes.  It  provides  the  requirements 
loop  analysis  in  the  SEP,  extends  the  user/stakeholder  JCIDS  involvement  through 
scenario  development,  provides  informed  architectural  trade-off  analyses,  and  vastly 
improves  the  software  developer’s  understanding  of  the  quality  requirements  in 
context.  Architectural  risk  is  significantly  reduced,  and  the  software  architecture 
presented  at  the  Preliminary  Design  Review  (PDR)  is  likely  to  have  a  much  higher 
probability  of  meeting  the  warfighters’  need  for  capability,  including  TOC  elements. 

Together,  the  QAW  and  ATAMsm  provide  effective  tools  for  addressing 
problem  areas  common  in  many  DoD  software-intensive  system  developments: 
missing  or  vaguely  articulated  performance  requirements,  significantly 
underestimated  software  development  efforts  (resulting  in  severely  underestimated 
schedules  and  budgets),  and  poor  communication  between  the  software  developer 
and  the  government  (both  user  and  PM).  Both  tools  provide  frameworks  for  more 
detailed  requirements  development  and  more  effective  communication,  but  they  are 
just  tools — by  themselves,  they  will  not  replace  the  need  for  sound  planning, 
management  techniques,  and  effort.  Both  the  QAW  and  ATAMsm  provide 
methodologies  for  executing  SEP  requirements  analysis  and  requirements  loop 
functions,  effective  architectural  design  transition  from  user  to  developer,  and  SEP 
design  loop  and  verification  loop  functions  within  the  test-case  development. 

A  significant  product  resulting  from  the  ATAMsm  is  the  development  of  test 
cases  correlating  to  the  use  case,  growth,  and  exploratory  scenarios  developed  and 
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prioritized.  Figure  2  depicts  the  progression  from  user-stated  capability  requirements 
in  the  JCIDS  documents  to  the  ATAMsm  scenario  development,  and  finally  to  the 
corresponding  test  cases  developed.  The  linkage  to  the  user  requirements  defined  in 
the  JCIDS  documents  is  very  strong  as  those  documents  drive  the  development  of 
the  three  types  of  scenarios,  and,  in  turn,  the  scenarios  drive  the  development  of  the 
use  cases.  The  prioritization  of  the  scenarios  from  user-stated  KPPs,  Critical 
Operational  Issues  (COIs),  and  FMECA  analysis  flows  to  the  test  cases,  helping  to 
create  a  system  test  program  designed  to  focus  on  effectiveness  and  suitability 
tests — culminating  in  the  system  Operational  Test  and  Evaluation  (OT&E).  FMECA 
is  one  of  the  focus  areas  that  will  have  a  dynamic  impact  on  TOC  analysis  because 
it  will  help  identify  software  components  that  need  higher  reliability  and  back-up 
capability.  The  MUIRS  focus  helps  ensure  that  TOC  elements  are  addressed  in 
design  and  test. 


JCIDS 

Docs 

Scenario  Development 

«^>| 

Test-case  Development 

Integrated 
into  test 
program 


Figure  2.  Capabilities-Based  ATAMsm  Scenario  Development 

The  traceability  from  user-stated  requirements  through  scenario  development 
to  test-case  development  provides  a  powerful  communication  and  assessment 
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methodology.  The  growth  scenarios  and  resulting  test  cases  are  particularly  suited 
for  addressing  and  evaluating  TOC  design  requirements  because  the  system 
evolves  over  its  life  cycle,  which  is  often  overlooked  in  current  system  development 
efforts. 


The  software  developer’s  understanding  of  the  eventual  performance  required 
in  order  to  be  considered  successful  guides  the  design  of  the  architecture  and  every 
step  of  the  software  development,  coding,  and  testing  through  to  the  Full  Operational 
Capability  (FOC)  delivery  and  OT&E.  Coding  and  early  testing  of  software  units  and 
configuration  items  is  much  more  purposeful  due  to  this  level  of  understanding.  The 
MUIRS  and  FMECA  focus  will  help  the  design  process  for  better  TOC  performance. 

The  resulting  test  program  is  very  comprehensive  as  each  prioritized  scenario 
requires  testing  or  other  verification  methodologies  to  demonstrate  how  the  software 
performs  in  each  related  scenario  and  satisfies  the  quality  attributes  borne  of  the 
user  requirements.  The  testing  supports  the  SEP  design  loop  by  verifying  that  the 
software  performs  the  functions  allocated  to  it  and,  in  aggregate,  performs  the 
verification  loop  process  by  demonstrating  that  the  final  product  produces  the 
capability  identified  in  the  user  requirements  through  operational  testing. 

Both  the  CAW  and  ATAMsm  require  the  capturing  of  essential  data  supporting 
decision-making  and  documenting  decisions  made.  These  databases  would  be  best 
used  in  a  collaborative  IT  system,  as  described  in  the  next  section. 

Collaborative  IT  Systems 

Collaborative  IT  tools  are  being  used  today  in  the  private  sector  to  connect 
various  stakeholders — designers,  logisticians,  cost  analysts,  field  service 
representatives  (FSRs),  system  users — who  have  the  need  to  communicate.  Such 
tools  could  be  used  to  support  current  and  emerging  warfighting  systems. 
Collaborative  tools  could  be  adapted  to  address  reliability  and  ownership  cost 
concerns  related  to  warfighting  systems.  Tools  that  facilitate  improved 
communications  would  likely  have  immediate  payoff  in  being  able  to  speed  up 
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solutions  to  problems.  For  example,  FSRs  and  users  could  quickly  raise  problems  to 
technical  staff  for  resolution.  Cost  analysts  could  more  quickly  identify  emerging  cost 
drivers  and  initiate  business  case  analyses  (BCAs).  Production  and  quality 
technicians  could  rapidly  learn  of  field  defects  that  are  the  result  of  production 
defects.  Other  FSRs  and  users  could  be  alerted  to  emerging  problems  and  be 
armed  with  advance  knowledge  that  might  avert  impending  failures. 

The  reliability  improvement  process  could  be  enhanced  by  the  use  of 
collaborative  tools,  because  of  the  ease  with  which  LCL  professionals  could  bring 
repair  parts  databases  to  bear  on  design  decisions.  This  would  be  helped  by  Pareto, 
that  is,  a  focus  on  the  cost  drivers  or  reliability  drivers,  especially  the  expensive 
items  that  fail  more  often  than  predicted.  This  approach  could  be  used  up  front  in 
pre-acquisition  phases,  too,  by  tying  in  legacy  databases  that  contain  performance 
information  of  similar  or  predecessor  systems. 

Think  of  the  impact  to  BCA.  Cost  estimates  depend  on  solid  cost  databases 
that  are  continually  updated  by  current  systems  in  order  to  identify  major  cost  drivers 
that  might  be  candidates  for  redesign  or  improved  manufacturing  processes  to 
achieve  better  reliability  and  reduced  LCC.  Collaborative  IT  could  contribute  to  the 
accuracy  and  completeness  of  cost  estimates. 

Component  improvements  that  result  from  collaborative  databases  would  pay 
off  in  legacy  systems,  but  might  deliver  a  second  payoff  in  reduced  ownership  cost 
of  future  systems  as  well.  Collaborative  databases  could  be  cross-referenced  in  an 
architecture  that  would  arrange  cost  and  reliability  information  in  system,  subsystem, 
or  component  databases,  enabling  better  cost  estimating  of  emerging  systems.  In 
her  2010  article  in  the  Defense  Acquisition  Review  Journal,  Marti  A.  Roper 
discussed  the  need  for  databases  that  support  acquisition  cost  estimates — down  to 
subsystem  or  component  levels,  showing  cost  ranges.  Such  a  knowledge  base  is 
critical  for  the  development  of  follow-on  systems  so  that  known  cost  drivers  can  be 
addressed  for  potentially  significant  LCC  savings  with  deployment  of  the 
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replacement  system.  Roper  referred  to  this  as  capabilities-based  parametric  data 
analysis  (2010,  pp.  71-73). 

An  example  of  the  potential  value  of  collaborative  efforts  in  improving 
reliability  and  reducing  TOC  is  the  microwave  tube  on  the  Aegis  program,  developed 
in  the  early  1980s.  The  tubes  were  expensive  to  maintain  (an  estimated  $8.20  per 
operating  hour)  and  ubiquitous  (nearly  30,000  units  in  2010),  and  initial  reliability 
numbers  were  lower  than  expected  (as  low  as  1 ,300  hours  mean  time  between 
failures  [MTBF]).  Through  a  collaborative  effort  between  the  PM,  NAVSEA,  and 
several  commercial  vendors,  design  and  manufacturing  improvements  increased  the 
MTBF  to  40,000-45,000  hours,  drastically  reducing  the  associated  TOC  from  $8.20 
to  $0.45  per  operating  hour  for  all  associated  Naval  combat  systems  (Apte  & 
Dutkowski,  2006,  pp.  3-21). 

Collaborative  IT  tools  could  potentially  be  implemented  through  apps  to  smart 
handheld  devices,  such  as  iPhones,  Androids,  or  Blackberries.  These  devices, 
which  are  ubiquitous  at  systems  commands  and  contractor  design  and  logistics 
facilities,  could  be  very  valuable  and  convenient  for  FSRs,  military  maintenance 
personnel,  and  even  users  in  some  environments. 

Very  possibly,  collaborative  IT  tools  are  in  use,  contributing  to  better  data  and 
faster  solutions  to  Service  member  problems  on  legacy  systems.  On  its  face,  the 
DoD  needs  to  embrace  such  tools  to  improve  the  flow  of  technology,  acquisition,  and 
logistics  information. 

Databases 

The  Defense  Acquisition  Management  Information  Retrieval  (DAMIR) — 
MDAP  Systems  database  is  a  “virtual”  repository  used  by  the  acquisition  community 
and  others  to  manage  MDAP  and  MAIS  systems  and  to  provide  relevant  information 
about  those  systems  across  the  DoD.  The  database  arrays  Selected  Acquisition 
Reports  (SAR),  Defense  Acquisition  Executive  Summary  (DAES)  reports, 

Acquisition  Program  Baselines  (APBs),  and  SAR  Baselines.  It  contains  other 
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program  information,  such  as  missions  and  descriptions,  system  performance, 
schedules,  cost  and  funding  (including  operations  and  support  costs),  Nunn- 
McCurdy  breaches,  contracts  performance,  and  manufacturing  and  deliveries.  The 
DAMIR  database  contains  some  capability  to  compare  programs  in  terms  of  cost 
and  schedule  performance  and  to  summarize  cost  and  schedule  information  (e.g., 
by  warfighting  system  or  Service). 

VAMOSC  databases  that  collect  O&S  cost  information  should  be  improved  or 
replaced  for  better  support  of  cost  estimating.  Current  GAO  reports  indicate  that 
VAMOSC  is  inaccurate,  incomplete,  and  internally  inconsistent.  VAMOSC  should  be 
able  to  provide  data  on  similar  or  predecessor  systems,  subsystems,  and 
components  in  support  of  programs  in  development,  in  addition  to  providing  accurate 
O&S  cost  performance  for  legacy  systems  in  their  sustainment  phase. 

Software  component  analysis  and  decision  databases,  like  those  that  would 
be  developed  using  the  QAW  and  ATAMsm  tools,  should  be  required  for  every 
software-intensive  system.  Software  continues  to  be  a  “wildcard”  in  estimating  both 
acquisition  costs  and  O&S  costs,  so  front-end  analyses  must  be  improved, 
cataloged,  and  shared  widely  through  a  collaborative  environment. 

Collaborative  databases  to  gather  enterprise/system/subsystem/component 
cost  information  should  be  established  to  facilitate  collaboration  among  experts  who 
are  widely  dispersed.  One  can  envision  collaborative  IT  systems  being  employed  by 
systems  commands  and  the  Defense  Logistics  Agency  (DLA).  Such  systems  could 
support  national-level  enterprise  requirements  at  one  end  of  the  spectrum  or 
components  at  the  opposite  end.  In  any  case,  collaborative  IT  systems  could  be  set 
up  for  broad  sharing  of  information  that  might  be  useful  to  developers  of  new 
systems,  to  maintainers  of  legacy  systems,  or  to  O&S  cost  analysts  trying  to  improve 
the  performance  of  components  that  are  cost  drivers. 
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Conclusions  and  Recommendations:  Major  Thrusts 
to  Control  TOC 


Many  of  the  TOC  initiatives  implemented  since  our  TOC  research  report  in 
2003  (Boudreau  &  Naegle,  2003)  are  definitely  steps  in  the  right  direction  for 
understanding,  assessing,  and,  ultimately,  reducing  the  TOC  financial  burden.  In  this 
research,  we  have  identified  several  areas  that  remain  as  significant  hindrances  to 
effective  TOC  assessment  and  reduction,  including  conflicting  policy  guidance, 
inadequate  or  missing  databases,  and  inadequate  process  controls  for  software  and 
SoS/net-centric  TOC  drivers.  Future  policy  and  guidance  should  address  these 
shortfalls  to  more  fully  address  TOC  issues. 

Controls 

Cost  Estimates 

The  DoD  has  not  yet  demonstrated  its  ability  to  estimate  program  costs  within 
reasonable  confidence  limits.  Estimation  of  developmental  costs  is  challenging  at 
best  and  is  not  yet  well  enough  supported  by  solid  cost  databases.  The  addition  of 
O&S  cost  requirements  makes  sense  from  the  perspective  of  life-cycle  affordability, 
but  again,  this  effort  is  not  supported  by  sufficient  O&S  cost  databases.  The 
development  of  SoS  and  net-centric  systems  exacerbates  the  cost-estimating 
problem  as  system-wide  changes  drive  platform  costs,  but  may  not  be  attributable  to 
the  platform  absorbing  the  cost.  Platform  changes  may  also  drive  system-wide 
changes,  again  driving  costs  that  are  not  attributable  to  the  system  level.  While 
these  costs  may  not  be  attributable,  we  recognize  that  they  still  need  to  be  tracked 
so  that  they  can  be  estimated  in  future  developments  and  so  that  root-cause 
analyses  can  be  applied  to  help  eliminate  the  sources  in  the  future. 

Certifications  at  Milestone  A  and  Milestone  B 

The  certifications  at  Milestones  A  and  B,  along  with  the  attention  of  the 
Director  CAPE,  undoubtedly  bring  attention  and  scrutiny  to  program  cost  estimates 
and  concerns  regarding  program  affordability  in  the  context  of  the  larger  warfighting 

ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY  -  33  - 

NAVAL  POSTGRADUATE  SCHOOL 


portfolio.  The  mandate  for  cost  certificates  is  a  major  improvement,  as  compared  to 
our  2003  research.  Cost  certificates  are  a  necessary  forcing  function  to  push  the 
DoD  toward  more  reliable  cost  estimating.  Again,  SoS  and  net-centric  system 
development  may  add  certification  challenges  as  the  associated  costs  are  typically 
not  foreseeable,  and  attributing  the  costs  to  a  specific  PM  may  be  difficult. 

Changes  to  Nunn-McCurdy  to  Include  an  O&S  Cost  Metric 

Unquestionably,  Nunn-McCurdy  requirements  have  become  more 
demanding  and  onerous.  As  challenging  as  acquisition  costs  (APUC  and  PAUC) 
are,  they  are  not  the  correct  metrics  when  viewed  from  a  life-cycle  cost  perspective. 
Nunn-McCurdy  metrics  need  to  evolve  into  measures  of  life-cycle  cost,  including 
O&S  cost  portion  (e.g.,  average  O&S  cost  per  system  per  hour  or  average  O&S  cost 
per  system  per  mile).  To  do  otherwise  is  to  encourage  poor  system  development 
choices  that  may  add  to  life-cycle  cost  rather  than  constrain  it. 

Mandated  Reviews 

Moving  the  PDR  Assessment  to  precede  or  coincide  with  Milestone  B,  as 
mandated  in  WSARA  (2009),  should  improve  decision-making.  That  is,  required 
warfighter  capabilities,  technological  maturity,  affordable  resources,  and  available 
schedule  must  be  compatible  with  the  system  specification  at  Milestone  B.  This 
cannot  be  properly  assured  without  completion  of  the  preliminary  design  because 
PDR  supports  preparation  of  resource  and  schedule  estimates.  To  that  end,  we 
recommend  that  software-intensive  systems  employ  the  SEI’s  QAW  and  ATAMsm 
process  tools  (or  similar-type  processes)  to  accomplish  the  following:  more  fully 
define  derived  and  implied  software-related  requirements;  improve  the  software 
developer’s  understanding  of  how  the  warfighters  use  and  maintain  the  system; 
understand  how  the  system  is  likely  to  be  changed,  modified,  or  made  interoperable 
over  its  life  cycle;  and  improve  the  developer’s  understanding  of  the  performance  the 
warfighter  expects  under  stressful  or  unusual  operating  scenarios.  These  process 
tools  should  vastly  improve  the  reliability  of  information  resulting  from  the  PDR  with 
regard  to  the  software  components. 
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Technological  Maturity 

The  Technology  Readiness  Assessment  (TRA)  Deskbook  (DUSD[S&T], 

2005)  was  published  in  2005  and  has  greatly  clarified  understanding  of  technological 
maturity,  yet  it  is  difficult  to  apply  to  software  development.  The  DoD  has  a  long  track 
record  of  moving  into  detailed  design  after  Milestone  B  without  the  necessary 
maturity  of  technology  to  complete  the  system  design.  The  result  is  almost  always 
program  delays  and  substantial  cost  growth.  Lack  of  technological  maturity  is  one  of 
the  major  causes  of  cost  growth  and  reflects  the  importance  of  Knowledge  Point  1 , 
as  described  by  the  GAO  (2003).  Because  software  development  defies  early 
maturity  estimation,  it  must  be  considered  separately  and  include  the  maturity 
evaluations  of  the  software  developer  (CMMI  or  equivalent),  as  well  as  the  maturity 
evaluations  of  the  materiel  developer/PM  (SA-CMM  or  equivalent). 

Today,  we  have  a  useable  template  to  discuss  and  reach  a  common 
understanding  of  technological  maturity;  we  know  the  importance  of  technological 
maturity;  we  have  a  mandated  certification — in  law  and  regulation — to  assure  the 
intersection  of  technological  maturity,  affordability,  available  budget,  and  schedule. 
The  DoD  knows  the  elements  of  knowledge  that  are  necessary  for  sound  decision¬ 
making  to  launch  development  of  a  new  warfighting  system.  This  also  applies  to 
COTS  or  GOTS  software,  but  software  development  depends  on  assessing  the 
maturity  of  the  developer  and  the  PM  office,  as  stated  previously. 

Navy  Gate  Reviews 

The  DoD  should  require  gate  reviews  for  use  by  all  the  Services.  Gate 
reviews  provide  for  oversight  and  governance  of  MDAP  life-cycle  cost.  These 
reviews  establish  a  process  to  bring  attention  to  ownership  cost  throughout  the 
developmental  cycle  of  warfighting  systems.  In  a  wider  sense,  gate  reviews  provide 
a  forum  for  lessons  learned  regarding  TOC.  While  emphasizing  affordability  through 
the  developmental  and  production  phases  of  individual  warfighting  systems,  gate 
reviews  provide  the  opportunity  to  balance  the  resources  provided  among  capability 
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portfolios,  and  potentially  to  assist  in  balancing  resources  across  all  of  the 
department’s  family  of  capability  portfolios. 

Configuration  Steering  Boards 

The  opportunity  to  grow  requirements  for  ongoing  programs  that  are  beyond 
Milestone  B  has  been  largely  taken  away  from  the  user  community  and  placed  into 
the  hands  of  each  Service’s  Configuration  Steering  Board.  This  is  likely  to  curtail 
major  cost  increases  in  programs  and  encourages  cost  reductions  based  on  PM 
recommendations  in  program  requirements  and  within  program  objectives. 
Congressional  language  on  changes  to  user  requirements  has  been  accommodated 
in  the  most  recent  version  of  DoDI  5000.02  (USD[AT&L],  2008),  dated  December  8, 
2008.  Implementation  of  this  guidance  entails  a  major  change  in  culture;  whether  it  is 
successful  in  reducing  ownership  cost  will  be  shown  overtime. 

Performance-Based  Logistics 

The  DoD  is  very  familiar  with  the  demands  of  sustainment — but  the  OSD  has 
not  insisted  on  proper  planning  and  implementation  of  affordable  sustainment.  The 
OSD  has  not  focused  enough  on  the  metrics  that  indicate  success  of  warfighting 
systems  or  on  the  cost  to  achieve  required  metrics.  Instead,  focus  has  been  on 
commodity  management,  with  the  DLA  being  a  prime  example,  where  metrics  have 
reflected  performance  of  the  support  organization,  but  not  weapon  system 
readiness. 

PBL  must  be  applied  more  widely,  such  that  non-PBL  systems  should  be  an 
unusual  occurrence.  PBL  requirements  initially  should  be  analyzed  vertically  by  an 
individual  system  such  that  the  warfighting  system  is  able  to  achieve  its  mission  and 
is  affordable.  However,  PBL  arrangements  also  should  be  analyzed  horizontally  to 
take  advantage  of  economic  quantities  and  other  efficiencies  that  might  be  provided 
by  using  common  support  systems.  PBL  metrics  also  should  be  devised  to  reflect 
the  individual  warfighting  system  (i.e. ,  vertical)  and  the  broader  support  system  or 
enterprise  (i.e.,  horizontal). 
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